Modern electronic systems are becoming more complex and market pressure demands that these complex systems are developed and brought to high-volume production within shorter timelines. These trends demand the development of tools and processes that create greater efficiency in electronics systems development.
A typical step in the process of bringing modern electronics systems profitably into production is performing functional verifications of designs. Functional verification is the process by which an electronics system design at some level of abstraction is tested to ensure that a design when manufactured will function according to the intent of the designer.
The functional verification process includes a process to achieve functional coverage closure. Functional coverage closure is achieved when the designer can prove that all of the intended verification goals (i.e., functional coverage goals) have been achieved by one or more test scenarios developed by the designer or some other engineer who understands the functionality of the electronic system.
Today's functional coverage closure is performed manually by an engineer that writes the coverage goals and then manually writes the tests based on their understanding of the design and how it will process the tests. The engineer also manually runs the tests, collects coverage goal hit/miss information, analyzes the resultant information, and then finally adjusts the tests in an attempt to achieve previously un-hit coverage goals in the next round of testing. This process is repeated until all required verification goals have been achieved.
Recent studies suggest that engineers spend approximately 70% of their effort on verification which has more than doubled from 30% within five years. Cost are rising for a variety of electronic systems and now even applies to FPGA-based electronic systems which in the past could be easily and inexpensively re-programmed after productization to resolve late-discovered defects.
U.S. Pat. No. 7,181,376 (the '376 patent) discloses an apparatus and method for employing coverage directed tests which includes a coverage directed test generation (CDG) engine that comprises a data analyzer which analyzes coverage data from a current test run of a test verification system and from previous test runs to determine which coverage events from a coverage model have occurred therein, at what frequency and which ones have not yet occurred; a coverage model listing coverage event which defines the goal of the test verification system; a Bayesian network which determines input data to the test verification system to achieve at least some of the coverage events; and a task manager coupled to the data analyzer and the Bayesian network which refers to the coverage model and queries the Bayesian network to produce input data to achieve desired coverage events. However, the system disclosed in the '376 patent relies upon supervised learning using a Bayesian learning based approach which does not scale as the complexity of the design increases. Moreover, the system disclosed in the '376 patent relies on purely random generation or human intervention to constrain the random generation of stimulus to feed its learning algorithm.
U.S. Pat. No. 7,331,007 (the '007 patent) discloses a method of harnessing machine learning to improve the success rate of stimuli generation. The '007 patent discloses a method for functional verification of an electronic systems design, comprising identifying a feasible space of initial state vectors that can enable generation of stimuli for the design; sampling the feasible space to obtain a sample pool of initial state vectors; generating test stimuli to stimulate the design using respective members of the sample pool; evaluating results of an application of the test stimuli; responsively to the step of evaluating, establishing a subspace of the feasible space, the subspace comprising favorable initial state vectors; and selecting new initial state vectors from the subspace for use in generating functional tests for the design.
Notably, the invention disclosed in the '007 patent requires a human to list constraints to feed into a Constraint Satisfaction Problem (CSP) based random stimulus generator. In order for the self-checking tests to be effective, the invention disclosed in the '007 patent assumes that given an assignment of the state vector, the generation success of any transaction is independent of any other transaction. Moreover, in order for the invention in the '007 patent to be effective, a system employing this invention needs a set of transactions that are comprehensive enough to serve as building blocks for the stimuli desired to be generated, an upper bound to the number of bits in the domain of each gate, an upper bound to the number of gates that form any of the Boolean functions, and a membership query (MQ) oracle capable of producing a correct label indicating successes or failure for any transaction given a possibly incomplete state vector assignment. Furthermore, the invention disclosed in the '007 patent requires a training set of samples which includes a sufficient amount of labeled data.
U.S. Pat. No. 7,849,425 (the '425 patent) discloses a method for generating a test case, the method comprising: generating, by a processor coupled to a test bench configured to execute a test plan on a design under test, a first test case for the design under test (DUT), the generating the first test case comprising traversing a first path through a hierarchy of goals from a start goal to an end goal; wherein a first parent goal traversed in a first level of the hierarchy of goals comprises a first definition of one or more of a slot and a method; and traversal of the first parent goal results in executing an action associated with the first parent goal and subsequently selecting a child goal of the first parent goal for evaluation; and assuming the first definition of the slot or the method by a child goal in a second level of the hierarchy of goals, if the child goal requires the slot or the method; wherein the assuming the first definition of the slot or method comprises searching the first path upstream to the first parent goal for the first definition of the slot or method. A system employing the invention in the '425 patent requires the user to create a graph to constrain the stimulus generation process.
U.S. Pat. No. 9,454,457 (the '457 patent) discloses a software test apparatus and a software test method and a computer readable medium thereof. The software test apparatus stores a software testing program, an under-tested code, a plurality of basic test benches and a plurality of candidate test benches. The under-tested code includes a hard-to-detect code and the hard-to-detect code has at least one hard-to-detect section. The software test apparatus runs the software testing program to execute a plurality of following operations such as parsing the hard-to-detect code to generate a condition-statement tree; based on the basic test benches and the condition-statement tree, using a support vector machine (SVM) to establish a support vector regression (SVR) predictor; and applying the SVR predictor to choose a best candidate test bench from the candidate test benches.
Notably, the invention disclosed in the '457 patent requires the use of a support vector machine (SVM), a strictly supervised learning process that leaves the system susceptible to the curse of dimensionality which states that in order for a learning algorithm to work, the amount of training data must grow exponentially with the addition of each feature or dimension of the problem. However, the increase in training data also increases the training time required for the algorithm, and at some point, when the dimensionality reaches a certain level, training cannot be feasibly achieved. The disclosed invention also requires parsing the hard-to-detect code without describing the manner in which this is done. Moreover, although the '457 patent discloses the use of plurality of candidate test benches, it fails to describe an effective way to generate these candidate test benches.
U.S. Pat. No. 9,454,467 (the '467 patent) discloses a method and apparatus for mining test coverage data. In particular, the '467 patent discloses a method of mining test coverage data, which includes at a device having one or more processors and memory, sequentially processing each of a plurality of coverage data files that is generated by executing the program using a respective test input of a plurality of test inputs, where the processing of each current coverage data file extracts respective execution counter data from the current coverage data file; after processing each current coverage data file, determining whether the respective execution counter data extracted from the current coverage data file includes a predetermined change relative to the respective execution counter data extracted from previously processed coverage data files. In response to detecting the predetermined change for the current coverage data file, including the respective test input used to generate the current coverage data file in a test input collection for testing the program.
The invention disclosed in the '467 patent also employs the use of a simple data mining technique that is not a learning-based approach to reaching coverage closure but is rather a manner to rate a list of pre-generated tests to achieve an efficient list of tests to run to reach a pre-determined coverage goal. Although the invention disclosed in the '467 patent claims to work best when each test input represents a respective unique combination of various parameter values, configurations, and input data, the '467 patent fails, however, to describe the manner to obtain this list of test inputs.
Resources required for chip design verification have increased tremendously over the last several years. This trend is expected to continue. As such, a need exists for an effective design verification system that meets the growing demand for future complex electronic systems. The present invention addresses this need.